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The purpose of this thesis is to develop a prototype safety information 
management tool to capture human error in Naval Aviation maintenance mishaps. The 
Human Factors Analysis and Classification System-Maintenance Extension aeee an 
effective framework for classifying and analyzing the presence of maintenance errors that 
lead to mishaps, incidents, and personal injuries, is the foundation of this management 
tool. The target audience for this information management system tool includes safety 
personnel, mishap investigators, Aircraft Mishap Board (AMB) members, and analysts. 
A review of three areas is needed to produce the prototype: (1) the collection, use, and 
management of accident information, (2) human error theories as related to aviation 
mishaps, and (3) the design of an effective mishap database tool. A usability study was 
conducted using potential end-users (Naval Aviation Safety Officers). The participants 
are given both written procedures to navigate through the prototype and an exit survey. 
The results of the survey, including objective and subjective responses about the 
prototype are gathered. The resulting data indicates an improved version of the prototype 
could directly lead to a decreased mishap rate and overall increased mission readiness 


due to the training and analysis opportunity it provides. 
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I. INTRODUCTION 

A. OVERVIEW 

(1) Background 

Naval Aviation has had great success in substantially reducing (by half) its Class 
A Flight Mishap (FM) rate in each successive decade between 1950 and 1999 (see 
Figure 1). Despite this achievement, the proportion of mishaps attributed to human error 
has remained at a relative constant rate of about four in five (Nutwell & Sherman, 1997). 
In 1996, a Navy F-14 Tomcat crashed shortly after taking off from Nashville, Tennessee 
killing both aircrew and three civilians on the ground (HFQMB, 1997). As aresult of 
this causes of this mishap being was solely attributable to human (aircrew) error, senior 
Naval Leadership established a Human Factors Quality Management Board (HFQMB) 
with the objective of reducing human error involvement in Naval Aviation Class A flight 
mishaps by 50 percent at the start of fiscal year (FY) 2000 (HFQMB, 1997). Aircrew 
human error was found to be a contnbuting factor in 60 percent of Class A FMs and, 
consequently, was the initial focus of the HFQMB. Although Naval Aviation had its 
lowest Class A FM rate in FY 1999, the HFQMB’s goal of reducing human error Piited 
mishaps by 50 percent was not achieved (N aval Safety Center, 1999). Thus, the scope of 
the HFQMB was expanded to include the reduction of human error in maintenance 
related aviation mishaps. Maintenance human error contributed to about 20 percent of 
the Class A FM rate (Naval Safety Center, 1999). This thesis contributes to this endeavor 
by developing an information management system to facilitate the characterization and 


analysis of human error in Naval Aviation maintenance related mishaps. 
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Figure 1: Naval Aviation Class A Flight Mishap Rates for FY 1950-1999 
(School of Aviation Safety, 1999) 


The HFQMB’s (1997) strategy to achieve its objective consists of a three-pronged 
approach: (1) Mishap Data Analysis (MDA), (2) Organizational Benchmarking (OB), 
and (3) Command Safety Assessment (CSA). MDA identifies human factors issues in 
past Class A FMs. Target areas were prioritized for intervention based upon the presence 
of prevailing human errors. The Human — Analysis and Classification System 
(HFACS) of identifying causal factors through examination of past mishaps was 
developed to achieve this end. OB is the second approach used to identify the best 
practices and procedures in other aviation organizations, both military and civilian. For 
example, the US Army attributed its reduced Class A FM rate to the use of Risk 
Management by its aircrew. Consequently, Operational Risk Management (ORM) was 
adopted by Naval Aviation and established as a part of doctrine (Department of the Navy- 
-DON, 1997). CSA was developed to assess a command’s safety climate from an aircrew 
perspective (Ciavarelli & Figlock, 1997). It solicits opinions/attitudes about 


organizational safety processes and command climate. CSAs may eventually show that 
Js 


both organizational and supervisory issues impact flight safety (Nutwell & Sherman, 
19973: 

In January 1999, the Vice Chief of Naval Operations revised the goal to reduce 
human error in Naval Aviation Class A FMs by 50 percent by the end of FY2000 
(Personal communication between T. Meyers and B. Goodrum, 1999). Presently one of 
every five Class A FMs contain maintenance error. This compelled the HFQMB to 
expand its focus to include maintenance related mishaps using the same three-prong 


approach as used for aircrew error analysis. 


(2) Maintenance Mishap Data Analysis (MDA) 

The analytic framework for examining aircrew and supervisory error in Class A 
FMs was HFACS. HFACS 1s a taxonomy which falls in line with the Naval Aviation 
Safety Program’s notion of multiple causal factors, the idea of sequential events leading 
to an event, and several established human factors theories. HFACS was adjusted and 
adopted to cover maintenance operations, and the extension was successfully used to 
examine major mishaps (Schmidt, Schmorrow, & Hardee, 1997), minor mishaps 
(Schmidt, Schmorrow, & Figlock, 2000), and ndidentainte (Schmidt, Figlock, & 
Teeters, 1999) data. The Navy has included an adjusted version of the Maintenance 
Extension of HEFACS (HFACS-ME) for inclusion in the upcoming revision of the Naval 


Aviation Safety Program Instruction (DON, 2000). 


B. PROBLEM STATEMENT . 

In order to continue to reduce its Class A FM rate, Naval Aviation leadership 
must understand that all mishaps are not caused solely by aircrew error. The analysis of 
maintenance related mishaps offers an increased opportunity to reduce target mishaps 
and enhance readiness. The HFACS-ME taxonomy has been adapted to classify causal 
factors that contribute to maintenance mishaps. A modern database tool is essential in 
more effectively addressing and identifying patterns of human error using HFACS-ME. 
However, there is no such tool available today. The target audience for such a tool would 
include safety personnel (e.g., data entry and retrieval by unit safety officers, other safety 
and training personnel, maintenance officers, maintenance supervisors), mishap 
investigators-for data retrieval (e.g., Aircraft Mishap Board members, squadron safety 
officers), and analysts (e.g., from the Naval Safety Center, the command’s safety officer 
or one from its higher headquarters). 

This thesis investigates the following questions: 

1. How could human errors in maintenance related Naval Aviation 
mishaps be effectively collected, cataloged, and collated in an information system? 

2. How could customers query and use this maintenance error information 
in order to identify problem areas and trends? 

3. How would customers in the fleet effectively and efficiently access 


maintenance error information in Naval Aviation mishaps? 


C. PURPOSE 

The intent of this study is to develop and evaluate a safety information 
management system that will facilitate data collection, organization, query, analysis, and 
reporting of maintenance personnel errors that contribute to Naval Aviation Tse 
equipment damage, and personnel injury using the HFACS-ME taxonomy as its basis. 

Drawing upon several theoretical approaches to examine mishaps involving 
human error, including Heinrich’s “Domino” Theory, Edwards’ “SHEL” Model, and 
Reason’s “Swiss Cheese” Model, helps to identify not only the unsafe actions causing a 
mishap, but latent conditions which set the stage for mishaps to occur. HFACS-ME is a 
composite derivative of these three taxonomies. It classifies and analyzes the presence of 
human error in maintenance operations leading to major mishaps, accidents of lesser 
severity, incidents, and maintenance related personal injury cases. However, working 
with a large database by hand or spreadsheet is very labor intensive. Given the capability 
of current relevant database tools, an improved information management system will 
bring HFACS-ME to the next level by improving access and analysis of safety data. 
Though there is no generally accepted method of accident investigation (Benner, 1975), 
standardized aircraft accident investigation — have been adopted by civilian and 
military agencies throughout the world (Diehl, 1991). 

The result of this study leads to a development of a tool that: (1) captures 
maintenance error associated with maintenance related incidents; (2) facilitates the 
identification of common maintenance errors and associated trends; and (3) supports 


understanding of how to identify human errors in the future. 


D. SCOPE AND LIMITATIONS ; 

Fleet personnel, primarily Aviation Safety Officers, will test the prototype 
Maintenance Extension Information Management System (MEIMS) tool (hereafter 
referred to as the prototype). The prototype to be developed is to be used by Naval 
Aviation squadrons, but may have some crossover use by other military branches and 
civilian airlines. Only maintenance related mishaps caused by human error will be 
considered. No material failure factors or maintenance related hazard reports or 


personnel injuries not related to a mishap are to be included. 


E. DEFINITIONS 

This study uses the following definitions: 

Aircraft Mishap Board. Group of officers appointed to investigate and report on 
an aviation mishap (DON, 1991). 

Aviation mishap rate. Number of aviation mishaps per 100,000 flight hours 
(DON, 1991). 

Aviation Safety Officer. Principal advisor to Naval Aviation squadron 
commanding officers on all aviation safety matters (DON, 1991) 

F-14 Tomcat. US Navy aircraft. Two aircrew, two engines, swing-wing, 
supersonic fighter with air-to-air, air-to-ground, and reconnaissance capability (Rowe & 
Morrison, 1973). 

Fleet Logistics Support Wing. US Navy reserve air wing comprised of transport 


aircraft (Schmidt, Schmorrow, & Teeters, 1999). 


HFACS: Human Factors Analysis and Classification System. System designed 
to help analyze Naval Aviation mishaps focusing on aircrew error (Shappell & 
Wiegmann, 1997). 

HFACS-ME: Human Factors Analysis and Classification —— 
Extension. HFACS adaptation to classify causal factors that contribute to maintenance 
mishaps (Schmidt, 1996). 

HFQMB: Human Factors Quality Management Board. Established by Naval 
Aviation senior leadership to reduce human error involvement in Naval Aviation Class A 
flight mishaps (HFQMB, 1997). 

MEIMS: Maintenance Error Information Management System. Prototype tool 
developed for this thesis. 

Mishap. A naval mishap 1s an unplanned event or series of events directly 
involving naval aircraft, which result in $10,000 or greater cumulative damage to naval 
aircraft, other aircraft, property, or personnel injury (DON, 1991). 

Mishap Categories. Naval aircraft mishap categories are defined below (DON, 
1991): 

Flight Mishap (FM). Those mishaps in which there was $10,000 or greater 

DOD aircraft damage or loss of a DOD aircraft, and intent for flight for DOD 

aircraft existed at the time of the mishap. Other property damage, injury, or death 

may or may not have occurred. 
Flight Related Mishap (FRM). Those mishaps in which there was less than 


$10,000 DOD aircraft damage, and intent for flight (for DOD aircraft) existed at 


the time of the mishap, and $10,000 or more total damage or a defined injury or 
death occurred. 

Aircraft Ground Mishap (AGM). Those mishaps in which no intent for 
flight existed at the time of the mishap and DOD aircraft loss, or $10,000 or more 
aircraft damage, and/or property damage, or a defined injury or death occurred. 
Mishap Severity Class. Mishap severity classes are based on personnel injury and 

property damage (DON, 1991): 

Class A. A mishap in which the total cost of property damage (including 
all aircraft damage) 1s $1,000,000 or greater; or a naval aircraft is destroyed or 
missing; or any fatality or permanent total disability occurs with direct 
involvement of naval aircraft. 

Class B. A mishap in which the total cost of property damage (including 
all aircraft damage) is $200,000 or more, but less than $1,000,000 and/or a 
permanent partial disability, and/or the hospitalization of five or more personnel. 

Class C. A mishap in which the total cost of property damage (including 
all aircraft damage) is $10,000 or more but less then $200,000 and/or injury 
results in five or more lost workdays. 

Naval Aircraft. Refers to US Navy, US Naval Reserve, US Marine Corps, and US 
Marine Corps aircraft. 

OPNAVINST 3750.6: The Naval Aviation Safety Program. US Navy instruction 
outlining Naval Aviation’s safety program. Revision Q-1991, revision R-in work (DON, 


1991 & 2000). 


ORM: Operational Risk Management. A decision making tool to increase 
effectiveness (and hence decrease accidents) by anticipating hazards, reducing the 
potential for loss due to these hazards, and thus increasing the probability of a successful 


mission (DON 1997). 


F. ORGANIZATION OF STUDY 

Chapter II contains a literature review on the development of a prototype to 
identify human error involvement and patterns in aviation maintenance mishaps. The 
methods used in this study are discussed in Chapter III. The results of this study are 
presented in Chapter IV. Finally, Chapter V contains conclusions, findings, and 


recommendations. 
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II. LITERATURE REVIEW 
A. OVERVIEW 

The literature examined relates to the development of a prototype to identify 
human error involvement and patterns in aviation maintenance mishaps. It memes 
textbooks, research articles, and masters theses pertaining to: (1) the collection, use, and 
management of accident information, (2) human error theories, its involvement in 
aviation mishaps, and specifically maintenance mishaps, and (3) design and usability of 
an effective mishap database tool. Collectively, these information sources provide a 
foundation to develop an effective and user friendly maintenance error analysis and 
reporting tool. 

Diehl] (1991), in a three-stage model of accident investigation and prevention, 
focuses on human performance and systems safety considerations (see Figure 2). The 
first stage is Accident Generation: the identification of hazards. Hazards have the 
potential to lead to an incident (near-accident) or even an accident. Heinrich (1941) 
study of thousands of accidents determined that for every major accident, there are 
approximately 30 minor accidents, and 300 hazardous incidents. This pyramidal 
relationship between hazards, incidents, and accidents also applies to aviation safety 


(Diehl, 1991). 
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Figure 2: Accident Generation, Investigation, and Prevention Elements 
(Adapted from Diehl, 1989) 


The second stage 1s the Accident Investigation Process: the collection, analysis, 
and review of accident data and the focus of this review. Accidents rarely result from a 
single sudden event, but are normally associated with a series of events degrading the 
performance of the equipment, crewmen, or both, until the accident is inevitable (Nance, 
1986). Investigating bodies have established similar aircraft accident investigation 
procedures. The fact-finding phase takes place near the scene of the accident to establish 
what happened. Next, the information analysis emphasizes on describing what caused 
the accident and why it occurred (Diehl, 1991). Part of that analysis is based on 
examining comparative data sources: information about both the normal and emergency 
performance of the aircraft, as well as human capabilities and limitations. Investigators 
are now able to theorize as to the causal factors of the accident and its probable sequence 
of events. Once the analysis is complete a final report is developed by board authorities 
with the accepted findings, causes, and recommendations. This phase is judgmental in 


nature. 
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The final stage contains the Prevention Measures (and methods) used to avoid 
future similar incidents. There are four categories of accident-prevention measures: 
(1) eliminating hazards and risks, (2) incorporating safety features (3) providing warning 
devices, and (4) establishing procedural safeguards. As one travels from n ste left 
along the bottom leg of Diehl’s triangle, the measures become less expensive, less 


effective, and less restrictive. (Diehl, 1991). 


B. ACCIDENT INFORMATION 

(1) Investigation 

Accidents occur within an organizational/systems context, and understanding the 
involved systems and operating environment can provide an enhanced framework for 
investigating accidents and determining their causes (Wagenaar, Groeneweg, & Hudson, 
1994). During the initial phase of an investigation retrospective analysis of past accidents 
can help to focus on areas of high risk and identify groups of potential causal factors 
(McElroy, 1974). Effective interventions can then be identified and subsequently 
implemented to reduce accident occurrence. However, the perceptions of accident 
investigators can be skewed and thus diminish the effectiveness of an investigation 
(Benner, 1982). Therefore, a systematic process for investigating and reporting accidents 
1s imperative. 

Grimaldi & Simonds (1984) detailed a four-part process for investigations. The 
first step is to explore the history of the incident as far back as is practical, including 
activities occurring both during and prior to the event. Second, the investigator must 


collect as many facts relating to the incident as possible (from reliable witnesses). Next, 
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the physical environment associated with the accident must be examined. Finally, the use 
of a defining guide listing common causal factors can be used to determine probable 
causal factors of the incident itself. This process parallels aspects of that provided by 
Diehl (1991) in his model of aviation accident investigation. 

Though there is no generally accepted method of accident investigation (Benner, 
1975), standardized aircraft accident investigation procedures have been adopted by 


civilian and military agencies throughout the world (Diehl, 1991). 


(2) Reporting 

Accident reports have generally centered on number of episodes and observations 
per unit time (Brown, 1990a). Frequencies and rates alone, however, do not provide a 
sound basis for understanding accidents (Brown, 1990a). The conventional process of 
reporting accidents by a description followed by supporting documentation varies in 
scope, depth, quality, objectivity, and contains inconsistencies and varying levels of 
completeness (Edwards, 1981). In addition, the traditional reporting format does not 
normally capture human factors information (Adams, Barlow, & Hiddlestone, 1981). To 
increase the usability of mishap reports, the information they contain must assist in the 
determination of cause and prevention of future accidents by ensuring collection, 
classification, and data recording methods are accurate and reliable. The usefulness of 
the reports is greatly increased when bias is removed and any future potential (based on 
frequency or severity) of occurrence is easily used (Adams & Hartwell, 1977). 

Three elements critical to the success of an accident reporting system are 


(Chapanis, 1996): (1) properly trained investigators, (2) a good accident reporting form, 
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and (3) a centralized facility for dealing with reports. If a mishap reporting system is 
unable to either prevent or reduce the severity of future accidents (Brown, 1990b) 
subsequent data analysis can be problematic (Pimble & O’Toole, 1982). Analysis of data 
from typical reporting systems has been accomplished through the following Sfitéss: 

e collecting data on past accidents within a population; 

e dividing the sample into groups with and without accidents; 

e obtaining measurements of individual characteristics on all participants; 

e statistically comparing the two groups; and 

e identifying any significant difference between the two groups, associating the 

differential characteristic with accidents. 
Using these methods has resulted in a more complete and thorough analysis effort. 

This general style of accident reporting has been used by many studies, but its 
methods have also been concluded to be suspect (Hale & Hale, 1972; Hansen 1988; Shaw 
& Sichel, 1971). The symptom may not actually be responsible for the accident, but may 
be related to another (correlative) variable which may, in fact, bear responsibility. 
Recently, accident reporting tools have been advanced and are supporting more rigorous 
and ordered methods of analysis (Leplat, 1989; Malaterre, 1990; Reason, 1990. The 
ability of a report to distinguish between causal and correlative variables determines its 


utility (Hill, Byers, Rothblum, & Booth, 1994). 
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(3) Data Management 

Once data pertaining to an accident is collected, it must be archived for use in 
accident prevention (National Safety News, 1975). Many methods of recording this 
information are in use, but some fundamental concepts are recognized including coding 
the data and the use of computer databases to quickly and efficiently store and retrieve 
information. In addition, the attributes of the data are critical in ensuring the best 
information is collected and stored for future use. 

The National Safety Council (National Safety News, 1975) established a method 
of facilitating the sorting and tabulation of accident particulars. Numerical codes are 
assigned to the different classifications in the mishap. Therefore, the specific case is read 
once when its facts are assigned code numbers. Concentrating on one phase of the 
accident problem at a time is a more effective way to reduce incidents than to deliberate 
on the mishap as a whole. Merely obtaining the information will not prevent recurrence 
of the accidents. The conditions contributing to the incident must be corrected. 
Subsequent sorting of these facts by category can then be completed quickly by simply 
referencing the code numbers. 

The Swedish Information System (ISA) on Occupational Accidents and Diseases 
was developed in 1979 to improve the work environment through increasing knowledge 
about risks (Andersson & Lagerlof, 1983). The ISA’s goal is accomplished through the 
collection of information in four areas: (1) knowledge about risks, (2) knowledge about 
preventive actions, (3) cost-benefit analysis, and (4) a “will” to change (see Figure 3). To 
identify a risk, the experience of one accident of a specific type is viewed as enough, and 


consequently accident prevention has traditionally been very case-related. However, 
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experience alone is not satisfactory for identifying and evaluating risks since minor 
incidents are frequently given less attention and major ones occur less frequently. The 
ISA system provides reports based on its database: periodic statistics, focused statistics, 
figures for particular accidents, and frequency of accidents. 

Knowledge 


Knowledge — about 
about risks preventive 


actions 


Improvement 


of work 
environment 


Cost-benefit “Will? to 
analysis change” 
Figure 3: Swedish Information System on Occupational Accidents and 
Diseases (ISA) (Andersson & Lagerlof, 1983) 





It is common to have a system that only reports incidents resulting 1n an accident 
(Grimaldi & Simonds, 1984). However, it is important to recognize “near-miss” cases in 
order to identify potential conditions or practices that are accident producing types and 
prevent their future occurrence. Essentially the same form could be used in both aoaiient 
and near-miss incidents. 

Setting up a computer analysis can reduce man-hours involved in reviewing 
mishap histories (Kuhlman, 1977). To set up an effective program, the available 
information needs to be organized and tabulated into categories. This information then 


can be presented in a format (report) to the user who can decide how to use it to analyze 


past occurrences. 
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(4) Prevention 

Interest in accident prevention did not begin until the beginning of the 20th 
century when employers realized that it was less expensive to prevent accidents than to 
pay for their consequences (Petersen, 1978). Organizations confronted with the challenge 
of how best to protect themselves and their employees from accidents have two options, 
namely, insurance and accident prevention programs (Pate-Comell, 1996), and 
organizations typically employ both options (Kanis & Weegels, 1990). 

Accident prevention was initially based on a widely held notion that people 
committing unsafe acts, not their working conditions, were to blame for most accidents 
(Heinrich, 1959). This thinking fostered a preoccupation with assigning blame to people; 
a practice which hindered the development of systematic accident prevention well into 
the later half of this century (Manuele, 1981). Narrowly focusing on people and not on 
the environment in which they operate, tended to obscure a subset of associated causal 
factors. This is particularly true with systems that chronically expose individuals to 
hazards (Schmidt, 1987). Although there have been substantial advances in accident 
prevention in recent decades, the practice of blaming individuals for the accident, rather 
than the conditions associated with it, persists. This practice must be overcome and 
accidents must be analyzed in terms of the systems 1n which they occur. 

The most effective accident prevention strategies employ systems engineering 
(Hawkins, 1993). The systems engineering approach was developed in the 1950s as part 
of the United States military’s large-scale weapons programs. Systems engineering 
transforms operational needs into a description of system parameters and integrates them 


to optimize overall system effectiveness (Edwards, 1988). In addition, it focuses the level 
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of analysis on the smallest identifiable system components and how these components 
interact (Bird, 1980). The strategy of focusing on the system through the development of 
well-defined system components exposes information that would have remained 
unknown without a system-level evaluation (Miller, 1988). | 

Systems engineering pays attention to the strengths and limitations of the human 
operator as an integral part of the system (Heinrich, Petersen, & Roos, 1980). The 
literature suggests that 80-90 percent of accidents are attributable to human error 
(Heinrich, Petersen, & Roos, 1980; Hale & Glendon, 1987; School of Aviation Safety, 
2000). Therefore, evaluating human factors associated with accidents can contribute to 
the understanding of systems and how they fail. 

Operational Risk Management (ORM) is another tool used by the armed forces to 
decrease aviation accident rates. It is a decision making tool used by personnel at all 
levels to increase effectiveness (and hence decrease accidents) by anticipating hazards, 
reducing the potential for loss due to these hazards, and thus increasing the probability of 
a successful mission (DON, 1997). The aviation arm of the United States Army achieved 
record low accident rates in 1995 and 1996 attributing a large portion of their success to 
the use of ORM (Department of the Army, 2000). The remaining military services 
institutionalized ORM in 1997 in attempt to lower their own rates (School of Aviation 
Safety, 2000). ORM emphasizes identifying hazards and reducing their associated risk to 
an acceptable level through the use of control measures (DON, 1997). It is especially 


effective in identifying and analyzing human factor hazards as well. 
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C. HUMAN ERROR 

Knowledge derived from the analysis of human errors can greatly improve safety. 
There are numerous theoretical approaches to examine mishaps involving human error 
(Goetsch, 1996). Some methods have their basis in industrial safety, while others are 
viewed from a more complex systems perspective, with an emphasis on human factors 
and operator error. Table 1 outlines some of the more well-known approaches. 


Table 1: Theoretical Approaches to Defining Accident Processes (Schmidt, 1998) 


Approach 


Industrial Safety Heinrich’s Domino Theory 
Systems Safety Edwards’ SHEL Model 
Reason’s Swiss Cheese Model 


(1) Heinrich’s “Domino” Theory 






The original accident causation theory is considered to be Heinnch’s “Domino” 
Theory (Goetsch, 1996). Heinrich believed accidents could be viewed as a linear five 
step sequence of related factors (chain of events) that lead to an actual mishap (Bird, 
1980). The two central principles of the Domino Theory are (Goetsch, 1996): (1) 
accidents are caused by the actions of the preceding factors, and (2) removal of the 
middle factor (unsafe act or condition) will negate the actions of the preceding factors 


and thus prevent accidents and injuries (see Figure 4). 
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Figure 4: Domino Theory (School of Aviation Safety, 1998) 


The following 1s a characterization of each domino (Bird, 1980): 


Lack of Control by Management (lack of supervision). The professional 
manager has four control functions: planning, organizing, leading, and 
controlling. Managers at all levels and all activities must perform these 
functions to ensure proper completion of work. 

Basic Cause(s) of incident--Ongin(s). A lack of management control (domino 
one) allows certain basic causes of incidents to exist. These causes are 
classified into two separate categories: 

o Personal Factors: denoted by a lack of knowledge or skill, improper 
motivation, physical or mental problems. Personal factors explain 
why people engage in substandard practices. 

© Job Factors: denoted by inadequate work standards, inadequate design 
or maintenance, inadequate purchasing standards, normal wear and 
tear, abnormal usage. Job factors explain why substandard conditions 


exist. 
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e Immediate Cause(s)--Symptoms (unsafe act/condition). If basic causes of 
incidents exist, the opportunity for actual substandard practices and conditions 
(errors) also exists. A substandard practice or condition is a deviation from an 
accepted standard or practice. 

e Incident--Contact. If substandard practices and conditions exist, an incident 
may occur that may or may not result in a loss. Of note, every incident that 
occurs provides an opportunity to collect data that could prevent a future 
occurrence. 

e Accident: Loss of People or Property. Once an incident has occurred it can 
result in a loss of personnel or property. 

Each step causes the next to occur, as would a series of falling dominos. If factors from 


any of the first three dominos are removed, the accident will effectively be prevented. 


(2) Edwards’ “SHEL” Model 
The “SHEL” Model (Edwards, 1988) was developed in the early 1970s to provide 
a more effective means to evaluate human-machine systems failures. The model 
identifies and classifies four dimensions in evaluating human-machine systems failures: 
Software, Hardware, Environment, and Liveware. 
e Software (S): rules, regulations, laws, orders, standard operating procedures, 
customs, practices, and habits that govern the manner in which the system 
operates and in which the information within it is organized; typically, a 


collection of documents. 
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e Hardware (H): buildings, vehicles, equipment, and materials of which the 
system 1s comprised. 
e Environmental conditions (E): operating setting (physical, economic, political 
and social factors) of the software, hardware, and liveware operate. 
e Liveware (L): people involved with the system. 
Thus the SHEL Model is comprised of these system dimensions and the 
relationships between them (see Figure 5). 


Software 


Hardware 


Environment 


Liveware 
Liveware 


Figure 5: SHEL Model of System Design (Hawkins, 1993) 





The SHEL Model assumes that a failure in the system will occur when one of the 
dimensions or the connection between them fails (Edwards, 1988). People are rarely the 
only cause of a mishap. They are, in fact, caused by the interaction of many factors 
(Shappell & Wiegmann, 1997). The SHEL Model is a significant change from the 
previously held idea that mishaps have single cause factors (Edwards, 1981). The SHEL 
Model describes systems, identifies areas for concern in a system, and provides a 


framework for accident investigation. 
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(3) Reason’s “Swiss Cheese” Model 

The Swiss Cheese model (Reason, 1990) of accident causation is another widely 
accepted perspective. Reason took a human factors approach to view the vertical 
association of a group of factors that lead to an eventual accident. His model 
differentiates between two error types: (1) active failures--the actions (or inactions) of 
operators that are believed to have caused the accident, and (2) latent conditions-- 
situations primarily caused by management decisions or actions whose repercussions may 
only become apparent when they are triggered by other mitigating factors. The 
conjunction between context and acts, when taken together, are latent conditions. Latent 
conditions set the groundwork for an accident while active failures are the final catalyst 
for the mishap to occur. Safeguards in a system can prevent latent conditions from taking 
effect by reducing the probability for the commission of an active failure. Thus Reason’s 
model seen as Swiss cheese slices’ lined in a row, with each vertical slice representing a 
defense layer and each hole representing an active failure or latent condition in the 


defense. An accident will occur when the holes are aligned (see Figure 6). 
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Figure 6: Reason’s Swiss Cheese Model (Schmidt & Lawson, 2000) 
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Reason’s (1997) model is not static, but dynamic, with each defensive layer 
moving according to the characteristics of the situation (Reason, 1997). An event may 
occur in one of three levels: (1) person-unsafe acts, (2) workplace-error provoking 
conditions, and (3) organization-error establishing conditions. The starting oe for an 
accident occurs with organizational factors; strategic decisions and associated processes 
(resource allocation, budgeting, forecasting, planning) are initiated. These 
organizationally established processes are shaped and influenced by a corporate culture 
being distributed throughout the organization to individual workplaces. Corporate 
processes evidence themselves as inadequate staffing, time pressures, equipment, 
training, and working conditions. These factors, combined with the natural proclivity to 
commit errors and/or violations results in unsafe acts. Very few of these acts actually 


create holes in the defense layers en route to becoming an accident (Reason, 1997). 


(4) Human Factors Analysis & Classification System (HFACS) 

A restructuring and expansion of the Swiss Cheese Model evolved into HEFACS 
which was specifically designed to help analyze Naval Aviation mishaps (Shappell & 
Wiegmann, 1997). HFACS focuses on aircrew error and also incorporates features of 
Heinrich’s Domino Theory and Edwards’ SHEL Model. The resulting taxonomy of 
unsafe operations identifies both active failures and latent conditions within four 
categories (DON, 2000): (1) unsafe acts; (2) pre-conditions for unsafe acts; (3) unsafe 
supervision, and (4) organizational influences. This classification can then be, and, in 


fact, has been, used to target the most appropriate intervention (see Figure 7). The Naval 
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Safety Center has adopted the use of the HFACS model for analysis of aircrew error in 


Naval Aviation mishaps. 
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Figure 7: Levels of Human-Component Failure (Schmidt, 1998) 

The following 1s a brief description of the HFACS taxonomy (DON, 2000). 
Unsafe acts take two forms: (1) errors and (2) violations. Errors are found in most 
mishaps due to the facts that human beings by their nature make mistakes and are often 
the last flaw before the mishap occurs. There are three basic error types: (1) decision, (2) 
perceptual, and (3) skill-based. Violations, on the other hand, are the willful disregard for 
the rules and are not seen in as many mishaps. They are also further categorized into 
routine and exceptional violation categories. Pre-conditions for unsafe acts set the table 
for the unsafe act to occur. Its two major subdivisions are (1) substandard conditions of 
operators and (2) substandard practices of operators. Substandard conditions included 
adverse mental and physiological states and physical/mental limitations. Substandard 
practices include crew resource management and personal readiness. Failures associated 
with unsafe supervision are divided into four areas: (1) inadequate supervision, (2) 


planned inappropriate operations, (3) failure to correct a known problem, and (4) 
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supervisory violations. Upper-level management failures, or organizational influences, 
directly effect not only supervisory practices, but operator conditions and actions. These 
latent failures can be traced to issues dealing with resource management, organizational 


climate, and operational processes. 


(5) Human Error in Maintenance Mishaps 

Maintenance level human factors in aircraft mishaps can be categorized similarly 
to aircrew level human factors (DON, 2000). The Maintenance Extension (ME) of the 
HFACS taxonomy was adapted to classify causal factors that contribute to maintenance 
mishaps (Schmidt, 1996). It contains four human error categories: (1) Supervisory 
Conditions; (2) Working Conditions; (3) Maintainer Conditions; and (4) Maintainer Acts 
(see Figure 8). These categories provide for multiple causations, a chain of inter-related 
events, and observation between the link between components providing for a combined 


approach to study human error and its causes. 
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Figure 8: HFACS Maintenance Extension (HFACS-ME) 
(DON, 2000) 
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Supervisory, Working, and Maintainer Conditions are latent conditions that can 
impact the performance of a maintainer (Schmidt, Schmorrow, & Hardee, 1997). This 
may contribute to maintainer act, an active failure, leading directly to a maintenance 
related mishap (MRM), maintenance condition, or personal injury. Thus, the HFACS- 
ME categories enable a safety analyst to identify failures at each of the four levels 
historically related to accidents (Schmidt, Schmorrow, & Hardee, 1997). The working 
conditions of a maintainer, as compared to those of the aircrew, will often play a more 
significant role in errors observed during maintenance evolutions (DON, 2000). 
Maintenance conditions have the potential to become a latent condition with which the 
aircrew would have to accommodate in flight and can also directly lead to mishap or 
injury through no fault of the aircrew. The three orders of maintenance error: first, 
second, and third order, reflect a decomposition of the error type from a macro to a micro 
perspective (see Table 2). 

The following describe the categories of the original HFACS-ME taxonomy. 
Supervisory Conditions may contribute to an active failure due to either unforeseen or 
squadron errors. Maintainer Conditions that can contribute to an active failure include 
medical, crew resource management, and personal readiness. (Schmidt, 1998) 

Working Conditions include the physical environment in which the maintainer 
works and the tools they use in the course of their work. Circumstances that can 
contribute to an active failure include poor environmental factors (lighting, weather, 
environmental hazards), inadequate equipment (damaged, unavailable, uncertified), and 


uncomfortable workspaces (confining, obstructed, inaccessible). (Schmidt, 1998) 
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Table 2: HFACS Maintenance Extension Categories (DON, 2000) 
Hazardous Operations 
Inadequate Documentation 
Organizational Inadequate Desi on 
Inadequate Processes 
Supervisory Conditions Inadequate Resources 
Squndren Inappropriate Operations 
Uncorrected Problem 
Supervisory Misconduct 
Unsafe Mental State 
Medical Unsafe Physical State 
Unsafe Limitation 
Inadequate Communication 
Crew Coordination Inadequate Assertiveness 
Inadequate Training/Preparation 
| Inadequate 
a Certification/ Qualification 
Personnel Readiness Infringement 
Inadequate Lighting/Light 
Environment . Unsafe Weather/Exposure 
Unsafe Environmental Hazards 
Damaged 
Working Conditions Equipment Unavailable 
| Dated/Uncertified 
Confining 
Workspace Obstructed 
Inaccessible 
Attention 
Memory 
Error Knowledge/Rule Based 
Skill Based 
Maintainer Acts Judgment/Decision Making 
Routine 
Violation —a 
Flagrant 
Sabotage 


Inadequate Supervision 
Inadequate Adaptability/Flexibility 
29 


























Maintainer Conditions 













Errors and violations are active failures in the form of Maintainer Acts. Active 
Failures can either directly cause damage and injury, or lead to a latent Maintenance 
Condition. Errors is substandard performance due to inattention, poor workmanship, and 
complacency. Violations are intended actions including both the routine or exceptional 
variety. Routine violations are consistent departures from rules and regulations condoned 
by management. Thus routine violations are considered to be acceptable departures from 
rules and regulations. Exceptional violations are substandard practices and actions not 
condoned by management. (Schmidt, 1998). 

HFACS-ME was effective in capturing the nature of and relationships among 
active failures and latent conditions present in 63 Class A (hull loss or fatality) Naval 
Aviation maintenance mishaps (Schmidt, Schmorrow, & Hardee, 1997), 470 reportable 
(over $10,000 damage or permanent/partial disability) Naval Aviation maintenance 
mishaps (Schmidt, Schmorrow, & Figlock, 2000), 124 incidents (Mishap Reports, Hazard 
Reports, and Injury Reports) for Fleet Logistics Support Wing maintenance operations 
(Schmidt, Figlock, & Teeters, 1999), and 15 select National Transportation Safety Board 
(NTSB) major (hull loss or fatality) maintenance accidents. The insight into latent 
conditions and active failures provides a solid perspective for trend analysis, investigation 


prioritization, and control development. 


(6) Maintenance Error Issues 
Marx (1998) stated that human error has not been served well by conventional 
accident investigation methods. These processes normally end once human error is 


identified without trying to understand why it occurred. This problem has been attributed 
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to several factors (Schmidt, 1998): (1) reporting criteria, (2) investigator biases, (3) report 
scope, depth, and quality, (4) reporting system design, and (5) database construction. By 
focusing on a human factors oriented investigation and reporting process, we can 
understand why people make certain mistakes. 

The prevention of accidents is critically linked to a sufficient investigation of 
human factors (Harle, 1994). Such investigation methods must be properly designed, 
implemented and supported. Past attempts at this have generated more superficial 
information than substance (Zotov, 1996) and have failed to properly consider the human 
element in the system (Bruggink, 1996). Human factors based investigation methods are 
considered by aviation industry personnel to be a better form of inquiry; however, they 
are not being widely used (Schmidt, 1998). Reason’s model established a conceptual 
framework of human error that has been widely accepted throughout government, 
military, and commercial sectors. Despite this acceptance, his model does not completely 


define the forerunners to accidents (Marx, 1998). 


(7) Maintenance Error Decision Aid (MEDA) 

Boeing Aircraft Corporation developed an event-driven tool to reduce 
maintenance related accidents by assisting investigators in the identification of accident 
contributing factors and recommendations for correction--Maintenance Error Decision 
Aid or “MEDA” (Hibit & Marx, 1994). MEDA supports human-centered error 
investigation in an attempt to encourage users to change their paradigm about 
investigations of maintenance error. MEDA is based on a maintenance system model 


where contributing factors are identified at each of four encompassing stages: (1) the 


3) I 


individual mechanic, (2) the mechanics immediate work environment, (3) the supervision 
provided to the mechanic, and (4) the organizational climate set for the mechanic 
(Boeing, 1997). 

Boeing collected in-flight shutdown (IFSD) due to maintenance error data from 
15 airlines with 500,000 to 1 million engine hours of Boeing aircraft between 1983 and 
1993 (Boeing, 1997). Each of these errors was assigned a causal factor before being 
added to the database. Success would be achieved by incorporating this system into an 
organization’s everyday operations with internal management of maintenance error 
providing the best return (Goglia, J., National Transportation Safety Board, personal 
communication with Schmidt, J.. 1998). MEDA is based on process improvement and 
discourages the practice of simply punishing the person who commits the error. 
Investigators establish contributing factors to the event and recommendations for process 
improvement, all of which are added to the MEDA database. Once improvements have 
been made, this information 1s provided to all affected employees (Boeing, 1997). 

Success has been cited by organizations using MEDA, e.g., reduction in 
maintenance related incidents, improved maintenance practices (Sargent & Smith, 1999). 
However, Marx (1998) notes that MEDA (and human factors based investigation 
methods in general) 1s not being widely used. Of 92 carriers using MEDA, only 6 are in 
the United States. Placing blame on workers, not transcending proximate causes, 
emphasizing the static who, what, when variables, not searching for underlying causes, 
and being only a philosophy vice an integrated solution were all cited as reasons for not 
using MEDA and other similar tools (Goglia, J., National Transportation Safety Board, 


personal communication with Schmidt, J., 1998). 
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Both Galaxy and BF Goodrich have created software applications for MEDA to 
transform it from a pencil and paper collection method to the information age. Galaxy 
developed ‘““TEAM”--Tools for Error Analysis in Maintenance 
(www.galaxyscientific.com, 2000) and BF Goodrich (BF Goodrich, 1997) Bliswea with 
a hybrid system that incorporates MEDA and another system called Aurora 
(www.hfskyway.gov, 1999). These applications allow the user to collect, organize, 
analyze, and report data through an interactive graphical user interface system. Users are 
able to enter new or update existing error data, create reports (e.g., MEDA forms, 
contributing factors/error summaries, and audit information/checklists), and update 


information on corrective actions being taken. 


(8) Sharing Maintenance Error Data 

The Flight Operational Quality Assurance (FOQA) program is a voluntary, but 
Federal Aviation Administration (FAA) sanctioned effort, to share aggregated safety 
data, continuous from flight data recorders, across commercial airline carriers 
(www.faa.gov, 2000). The intent is to provide a means to examine industry wide trends 
and use the derived information to enhance training of personnel, operational procedures, 
maintenance and engineering, air traffic control, and airport surface safety. FAA 
Admiunistrator Jane Garvey stated, “FOQA programs are already producing the hard data 
we need to identify safety records, target potential problems, and make corrections before 
accidents happen (Reuters, 2000).” Data to be collected includes Ground Proximity 
Warning System (GPWS) warnings, excessive rotation rates on take off, un-stabilized 


approaches, hard landings, and compliance with standard operating procedures. 
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Additional information includes fuel efficiency, out-of trim airframe configuration 
identification, engine condition, compliance with noise abatement, rough runway 
surfaces, and aircraft structural fatigue (Garvey, 1998). 

Presently, 230 total aircraft consisting of 13 aircraft types are electronically 
collecting/sharing FOQA data (Reuters, 2000). An impetus for sharing under the banner 
of safety is that shared FOQA data is not used for enforcement purposes except under 
egregious circumstances. This cooperation has not been as successful in extending to the 
hangar bay and flight line in terms of maintenance and sharing error and incident data. 
The FAA and NTSB both concur that this is an essential part of the overall safety 
equation for increasing commercial aviation safety. One major problem standing in the 
way is having a common process/taxonomy for capturing, recording, and archiving 
accident/incident/error data for aggregate and trend analysis (Goglia, J., National 
Transportation Safety Board, personal communication with Schmidt, J., 1998). 

Boeing’s MEDA, Galaxy’s TEAM (Tools for Error Analysis in Maintenance) tool 
and BF Goodrich’s MEDA software tool all attempt to achieve a vehicle for not only 
capturing mishap information, but also to share data across the industry (Goglia, J., 
National Transportation Safety Board, personal communication with Schmidt, J., 1998). 
Unfortunately, though used by some of Boeing’s customers (e.g., BF Goodrich in its re- 
work facility), MEDA has not been adopted as an industry standard (Marx, 1998). This 
is due, in part, to the in house requirement to staff such an initiative, the training 
requirements involved, and issues related to unions, culpability, etc. The latter is tied to 


the emphasis on the immediate act of the person and not the organizational and work 
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settings that have contributed to it. Consequently, a need exists to develop a tool 


encompassing accident data collection, organization, analysis, and reporting. 


D. TOOL DESIGN CONSIDERATIONS 

(1) System Design 

The usability of a product is directly linked to the user interface. A user interface 
that is easy to learn and use will have favorable usability evaluations. To maximize the 
usability of an interface, Shneiderman (1997) proposed eight golden rules of graphical 
user interface design: (1) consistency, (2) shortcuts for frequent users, (3) informative 
feedback, (4) dialogs designed to yield closure, (5) error prevention and simple error 
handling, (6) simple reversal of actions, (7) internal locus of control, and (8) reduced 
short-term memory load. A sense of comprehension and competence among the users 
will be the end result of following these rules. If users feel familiar and competent with 
systems, they will more likely them highly. (Shneiderman, 1997). 

Consistency can relate to many aspects of the system: terminology, color, layout, 
input, display formats, etc. Though consistency cannot always be maintained across all 
dimensions of a system, symbology and oe of interaction should be consistent. 
Frequent users can reduce the number interactions required for a specific result through 
shortcuts. These will also increase the pace of interaction. /nformation feedback can 
vary in degree depending on the frequency and severity of action involved and allow 
users to more fully understand their current status by immersing them in the graphical 
environment. Designing dialogs to yield closure can be achieved by grouping actions to 


set up a natural flow through the user’s tasks. This gives the user a better sense of the 
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actions as they are being performed and a better awareness of a sequence’s closure. 
(Shneiderman, 1997). 

Error prevention/handling allows the user to maintain confidence in the system’s 
fidelity and ability to recover from minor fluctuations. Reversing an action allows users 
to recover from their mistakes easily, reducing stress or anxiety of operating within the 
system. Designing an internal locus of control allows the users to be in command of the 
environment and not vice versa. Users would not experience autonomous movement 
within the environment or a drastic change of the visual orientation. Finally, the 
reduction of short-term memory load includes access to integrated assistance information 


(e.g., cues, mnemonics, standardized sequence of actions). (Shneiderman, 1997). 


(2) Usability Study 

Usability testing is a systematic means of observing the ease of use of a product 
and collecting related data. Dumas & Redish (1994) identified three tenets of usability 
studies: (1) It should be used to diagnose problems vice determining that the product is 
flawless; (2) Usability testing should be employed early and often during the 
development cycle; and (3) It is part of a process that focuses on usability throughout 
design and development. 

A thorough testing plan needs to be developed in order to best incorporate 
usability into the development process. Several factors must be addressed in an 
evaluation plan (Shneiderman, 1997; Nielsen, 1993; Hix & Hartson, 1993; Preece, et. al., 
1994; Newman & Lamming, 1995). First, the current stage of the design will determine 


the requirements for testing, with different conditions for different stages. Second, the 
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criticality of the environment helps to decide the objectives of the test. Finally, other 
factors, such as the novelty of the project, number of expected users, time available, cost 
of the project, available resources (e.g., time, money, people), and experience of the 
testers all play a role in defining the usability study. 

A usability study not only maximizes the usability of the system, but ensures 
contractual requirements have been completed and to provide evidence of testing in cases 
of lawsuits or if legal issues arise. Errors in a system will be tolerated to varying degrees 
dependent upon the need to bring the system to full operational use and the impact the 
errors may have during that time. However, a system is more difficult to test as 
increasing amounts of input are required, yet these tests are increasingly needed. 
(Shneiderman, 1997). 

Some (Nielsen & Mack, 1994) argue for an expert evaluation of a system to 
increase a product’s usability. Formal reviews can provide more useful information 
when compared to informal demonstrations of a product. Thus, system design and 
testing should employ expert reviewers who typically produce a report detailing problems 
and recommendations for improvement. These reviews may take the form of heuristic 
evaluation, guideline review, consistency inspection, cognitive-walkthrough, and formal 
usability inspection (see Table 3). Even if the experts are reviewing unfamiliar systems 
and technologies, they still provide a fresh look at a system and are useful in evaluating 


system development. 
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Table 3: Expert Review Methods (Shneiderman, 1997) 


Expert-Review Description 
Method 


Expert reviewers critique an interface to determine conformance 
with a short list of design heuristics such as the eight golden rules. 
The interface is checked for conformance with the organizational 
Experts verify consistency across a family of interfaces, checking 
for consistency of terminology, color, layout, input and output 
formats, within the interfaces as well as in the training materials 
and online help. 
Experts simulate users walking through the interface to carry out 


oa typical tasks. Simulating the day 1n the life of the user should be 
walkthrough 
part of the evaluation. 


Formal usability |Experts hold courtroom-style meeting, with a moderator to judge, 
inspection to present the interface and to discuss its merits and weaknesses. 


Shneiderman, 1997 











Heuristic evaluation 


Guideline review 


Consistency 
inspection 































Usability studies take additional forms, such as discount usability engineering; 
“short and sweet” approaches to task analysis, prototyping, and testing. Another style of 
study is a field study conducted in actual work environments in order to achieve realistic, 
user evaluation. Alternatively, beta testing challenges actual users break the system. 
This method can expedite the development process and correct errors missed through 
conventional testing. Usability testing can lack comprehensive system evaluation due to 
time constraints and also tends to emphasize first-time usage (Shneiderman, 1997). Thus 
usability studies must be supplemented with other methods of evaluation, such as expert 
review. (Shneiderman, 1997). 

An important decision to be made when planning a usability study is how long the 
test should take (Dumas & Redish, 1994). If the study is conducted as an integrated part 
of the design process and is not simply being conducted on a completed system, then the 
test length should be reduced to a level to both obtain necessary information and not be a 


burden. Dumas & Redish (1994) place traditional testing into four categories. 
38 


(1) Formal testing with comprehensive test reports requires eight to twelve weeks. (2) If 
a strong collaboration is exhibited among team members and a shortened report format is 
used, then four to six weeks are required. (3) If a particular part of the system is to be 
studied with well established procedures, then one week may be appropriate. (4) Finally, 
just-in-time testing can provide useful information in a few days, if necessary, but is 
discouraged. 

Dumas & Redish (1994) contend proper planning includes defining goals, 
identifying concerns, recruiting individuals and their participation, creating and 
organizing tasks/task scenarios, deciding on usability measures, preparing test materials 
and test environment, and conducting a pilot test. Defining goals and identifying 
concerns 1s a three-stage process: (1) making choices among goals and concerns; 

(2) moving from general to specific concerns helping to mold concerns into quantitative 
objectives; and (3) understanding the source of the goals and concerns allowing the 
usability engineer to better develop the testing scenarios and tasks. Developed user 
profiles, preferably prior to system design and usability testing, can provide the basis for 
deciding upon participants in a study. The realities of time and budget constraints often 
result in usability studies having less participants than usability engineers (10-12) or 


Statisticians (at least 36-48) may desire. (Dumas & Redish, 1994). 


(3) Human-Computer Interface (HCI) Design Issues 
Brown (1989) posits that information database management system should be 
considered a simple tool, simplifying rather than complicating the tasks of the user. Thus 


the design of the system should reduce mental processing operations (learning complex 
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commands/syntax, memorizing codes/abbreviations, translating data into other 
units/formats) required to operate the tool. He feels the allocation of functions, one of the 
most important categories of design decisions, should be performed based on the 
capabilities of both the user and the system. Brown (1989) suggests five “user” rules for 
allocating functions in routine interface design: (1) Minimize the amount of procedural 
memorization. (2) Reduce mental manipulation. Data should be presented clearly and in 
a useable form. (3) Minimize manual entries. Allow the user to select from a displayed 
list vice forcing manually entries. (4) Offer computer aids (e.g., checklists, summary 
displays, and help functions) to reduce both required mental processing and the need for 
executing complex, multi-step procedures. (5) Use computer algorithms to process and 
present complex data. 

Mental models, or a cognitive representation of the internal parts and operations 
of a system, are another important part of successful HCI. The user’s mental model 
allows him/her to predict the appropriate procedure for a desired outcome, even if the 
procedure has been forgotten. Thus, a user’s mental model can give the user an 
understanding of how a system works and develop and refine his/her knowledge when 
learning about or using the system. Several principles for mental models should be 
incorporated into a system: consistency (both actions and classes of actions), physical 
analogies, user expectations, and stimulus-response compatibility. (Brown, 1989). 

The designer should provide for a balance of ease of learning, ease of use, and 
functionality in a system. Brown (1989) identifies four techniques to ensure this balance 
is maintained: (1) incorporate needs of novices, experts, and intermittent users into the 


design, (2) avoid excess functionality, (3) provide multiple paths, and (4) progressive 
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disclosure and graceful evolution design. These procedures will ensure equilibrium is 


established in a system. 


E. SUMMARY 

A well-defined, systematic accident investigation, analysis, and reporting process 
is critical in the effort to reduce the Naval Aviation mishap rate. Yet, no one good 
universal system currently exists (Marx, 1998). Such a tool must have a reporting system 
relying on a solid data collection process with the ability to readily access the stored data. 
Many times, mishap data is lost or not used, often leading to yet another incident. The 
goal for such a tool is to use the data for prevention of future accidents. 

Effectively addressing human error issues can greatly increase safety levels. 
Several robust approaches to examining mishaps involving human error achieve this 
goal: Heinrich’s “Domino” Theory, Edwards’ “SHEL” Model, and Reason’s “Swiss 
Cheese” Model. Once an approach is examined, a means of bridging the gap between 
theory and user must be made. The Navy’s Human Factors Analysis & Classification 
System (HFACS) and its Maintenance Extension offshoot (HFACS-ME) appear to be 
potential approaches for investigating, reporting and analyzing maintenance error. 

However, the designer must maximize the usability of the interface. This can be 
accomplished by following Shneiderman’s (1997) eight golden rules of graphical user 
interface design. Once the system is designed, a usability study with a prototype tool 
ensures the product is ready for general use by testing it with a selected group of users. 


Finally, Human-Computer Interface issues must be addressed to simplify rather than 


4] 


complicate the life of the user. Once the above goals are met, the system is ready to meet 


its challenge. 
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I. METHODS 

A. RESEARCH APPROACH 

A software desktop analysis and reporting tool for maintenance error in aviation 
would greatly facilitate Naval Aviation’s effort to capture human factors in sea. and 
develop interventions. The Maintenance Error Information Management System 
(MEIMS) is a computer-based prototype tool designed using Microsoft Access 97 and 
Visual Basic 6.0. The prototype utilizes a database derived from the Naval Safety 
Center’s Safety Information Management System (SIMS) database, which contains 
information on over 600 maintenance error related mishaps that occurred between 1989 
and 1999. The system has a graphical user interface (GUI) that allows the end-user to 
operate the system with basic computer skills. 

The prototype was distributed to a representative sample of potential end-users. 
The participants were provided a prepared task list that required them to navigate through 
and utilize features of the tool. At the completion of the task list, the participants had 
viewed and used all portions of the prototype tool, and completed an exit survey 
composed of questions pertaining to demographic background information and both 
objective and open-ended items to elicit the participants’ views of the usability of the 
system and value of both the system and the data. The objective data was transcribed into 
a Microsoft Excel spreadsheet for analysis while a content analysis was conducted on the 
open-ended survey questions. Note, the exit survey used only five Likert style questions 
because the major focus of the effort was the creation of the prototype tool vice the 
usability study. The questions were shaped intuitively and are considered to be simply 


the first stage of developing a formalized post-prototype tool. 
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B. DESCRIPTION OF MAINTENANCE EXTENSION INFORMATION 
MANAGEMENT SYSTEM (MEIMS) 


(1) Overview 

The Maintenance Extension Information System (MEIMS) prototype tool was 
designed to allow the user to have access to the data base via three functional tools: (a) 
sorting the data by queries, (b) obtaining output from the data base via written reports 
and graphical displays, and (c) providing input to the data base through the addition of 
new data. Each function was displayed on separate pages with interactive controls 
providing user-prototype interaction. The following paragraphs provide a description of 


the prototype. A completion description of all of the displays is found in Appendix A. 


(2) Main Menu 

The Main Menu of the prototype allows the user to select (left click) one of five 
different options: (a) Query Menu, (b) Report Menu, (c) Expert Graph Menu, (d) Adding 
New Data, and (e) Exiting the system (see Figure Al). Help is provided to the user on 
this and all pages in the form of “control tips” (1.e., brief descriptions) when the mouse 
arrow is placed over a control (1.e., text box, command button, etc.) (see Figure A5). 
Additional help is found in the “status bar” at the bottom of the screen when a control is 


highlighted. 


(3) Query Menu 

The Query Menu provides the user two manners of output (see Figure A2). The 
first is through the selection of one of eight command buttons to sort the data base by one 
or more of its fields: aircraft model (F-14, H-46, etc.), aircraft type (tactical aircraft 
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(TACAIR), helicopters, heavy aircraft, trainers, and others), branch of service of the 
aircraft (USN, USMC), location of the mishap (ashore, embarked, and detached), mishap 
classification (A, B, or C), mishap type (Flight Mishap (FM), Flight-Related Mishap 
(FRM), or Aircraft-Ground Mishap (AGM)), calendar year of the mishap (1 989-1999), 
and any combination of the above (Multiple Criteria selection). 

When a single category control is selected a sub-menu appears where the user can 
define the exact description of the category via a combo box (see Figures A3 and A6). 
Upon selecting the “View Selection” control a Maintenance Mishap Query window is 
displayed revealing each instance (mishap) of the selected description (see Figure A4). 
In addition, the user may page through all mishaps of the selection by selecting the nght 
arrow on the bottom of the window (see Figures A4 and A5). The data for each mishap 
is displayed in text boxes, with the selected category denoted with blue background (see 
Figure A4). Additionally, maintenance related contributing factors to the mishap with 
their HFACS-ME codes are displayed at the bottom of the window. Selecting the 
“Multiple Criteria” control on the Query Menu will allow the user to further define the . 
data base by selecting any or all of the seven solo categories (see Figures A7 and A8). A 
Multiple Criteria sub-menu appears and allows the user to “check” the desired categories 
and further define them by selecting criteria provided in combo boxes on the sub-menu. 
In these cases, all of the selected categories will have a blue background on the resulting 
Maintenance Mishap Query window (see Figure A9). 

Throughout the prototype, “Define HFACS Codes” controls are displayed. When 
selected they will provide a summary sheet of the level one, two, and three HFACS-ME 


codes, with each acronym defined (see Figures A9 and A10). In addition, at any time the 
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user may close a window and return to the previous sub-menu by selecting the “Close 
Form” or “<Back” control (see Figure A5). Each of the four primary menus has a 
‘Return to Main Menu” control which returns the user to the Main Menu when selected. 

The second output provided by the Query Menu is the HEACS-ME Summary 
Form (see Figure All). This form allows the user to view a summary of data categorized 
by HFACS-ME levels one, two, and three. The user may also further define the output 
by selecting any or all of the previously mentioned seven categories via combo boxes 


(see Figure A12). 


(4) Report Menu 

Report Menu is the second option a user may select from the Main Menu (see 
Figure Al3). The Report Menu offers eight reports which provide data listing the total 
number of mishaps and the number and percentages of mishaps by HFACS-ME levels 
one, two, and three (see Figure Al5). The user may select from the following 
distribution presentations: all mishaps, aircraft model, mishap class, mishap type, mishap 
class by mishap type, branch of service, mishap location, and chronological listing by 
aircraft model (see Figures Al4 - A16). All reports are closed by selecting the “Close” 


control at the top of the window. 


(5) Expert Graph Menu 
The third option from the Main Menu 1s the Expert Graph Menu (see Figure 
Al7). The user may create a two-axis, three-dimensional graph presentation. The x- and 


y-axes are populated with one of the following categories: aircraft model, aircraft type, 
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mishap location, branch of service, mishap class, mishap type, HFACS-ME level one, 
HFACS-ME level two, and HFACS-ME level three (see Figure A18). The user may then 
select one or more of the fields from each axis category sub-menu via a combo box (see 
Figures A18 and Al9). The resultant graph is presented in a dnozranChsion’l: multi- 


colored view (see Figure A20). 


(6) Add New Data Menu 

The user may populate the data base by selecting the “Add New Data” control on 
the Main Menu (see Figure A21). The user must then fill in an “Enter New Maintenance 
Mishap Data” form with the following fields: mishap class, mishap type, date of mishap, 
aircraft type/model (F-14, H-46, etc.), aircraft category (TACAIR, helo, etc.), branch of 
service, location of mishap, and a brief description of the mishap. The prototype 
automatically assigns a Mishap Identification Number. A sub-menu on the form asks the 
user to input mishap “‘factor’” data. This information includes: a brief description of the 
factors and the HFACS-ME level three code. Upon selection of the level three code, the 
levels one and two codes and descriptions and level three descriptions are automatically 
entered by the prototype, as is the Factor Identification Number (see Figure A23). The 
user can enter an additional factor by selecting the “Add New Factor” control (see Figure 
A23). After all mishap data is entered, the “Close Form” control is selected (see Figure 
A23). The “Final Data Entry” form appears and asks the user to select “Enter” and wait 


for the ‘““Done”’ box to be checked to show successful data entry. 
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C. DATA COLLECTION ; 

(1) Subjects/Participants 

Students (n=42) attending the Aviation Safety Officer (ASO) course at the School 
of Aviation Safety, at the Naval Postgraduate School in Monterey, CA participated in the 
study. Individual activities are allocated class spaces to determine enrollment in the ASO 
course and, consequently, attendee demographics represent a wide cross section of Naval 
Aviators and Naval Flight Officers, Coast Guard and other DOD officers, Flight 
Surgeons and Aeromedical Safety Officers, and foreign nationals from all aircraft 
communities. ASO course graduates are responsible for the management and 
implementation of squadron safety programs to include mishaps and include 
investigations and reporting. They are likely to be one of the primary end-users of the 
tool. Participant demographics were characterized by aviation background, computer 
experience, and availability of software and hardware systems used in the Navy and 


Marine Corps. 


(2) Apparatus 

The ASO students had access to three computer labs (Pentium level) at the School 
of Aviation Safety via login identification and password to a group account. The 
computer in each lab had a full prototype functioning desktop analysis and reporting tool 
loaded onto it. After a participant gained access to the computer, the “MEIMS” icon was 
found on the computer desktop and selected to open the application. 

The prototype was developed using Microsoft Access 2000 and Visual Basic 6.0 


and consisted of four sections: database queries, reports, graphic presentations, and data 
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entry. Each section was divided into further subsections allowing the participant to more 
specifically design his access to the database. This allowed the participant to achieve the 
four functional requirements for the software tool: data collection, organization, analysis, 


and reporting (see Appendix A for a more complete description of the prototype). 


(3) Instrument 

A participant usability survey was constructed by the author consisting of three 
parts: (1) Participant demographics, (2) Likert type assessment questions, and (3) Open- 
ended items. Collection of demographic information was accomplished through the 
participant selecting from a list of descriptors (rank, branch of service, experience 
level/years of service). Survey questions were designed to determine if the prototype 
software tool met participant investigation, reporting, and analysis requirements. The 
Likert questions used a five point rating scale with verbal anchors: Strongly Agree, 
Agree, Neutral, Disagree, and Strongly Disagree. Open-ended questions were included 
to gain subjective responses on the overall impression of the prototype software tool, 
recommendations for improvement, and comments on areas not adequately covered by 


parts one and two of the survey (see Appendix B). 


(4) Procedure 

The prototype software desktop analysis and reporting tool containing a database 
derived from Naval Safety Center maintenance mishaps was loaded on seven computers 
in three computer labs with 24-hour accessibility. A MEIMS icon was placed on each 


computer desktop page to allow access into the program. 
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Prototype testing occurred over a span of one week. Participants were given a 
group onentation briefing by the author on the purpose, goals, and procedures of the 
prototype including a projected computer demonstration. In addition, they were given a 
10-page guide to walk them through prototype testing (see Appendices B and C). The 
guide consisted of: 
e Instructions for Accessing the Prototype Software Tool --information to log 
on and open the prototype (see Appendix B). 

e Prototype Software Tool Task List--a series of planned navigation routes 
within the prototype whereby the participant would be able to view the entire 
system (see Appendix B). 

e Participant’s Impression of the Prototype Software Tool/Exit Survey 
(see Appendix C). 

The author, with full knowledge of both prototype MEIMS tool and Microsoft 
Access procedures took six minutes to complete the testing. It was expected that each 
participant would need 15-20 minutes to complete the process. Though information on 
time to navigate for each individual was not taken, informal feedback to the author 
indicated a range of 15-30 minutes with the longer times being needed for those with less 
computer and Microsoft Access experience. One computer was a lower-end model (1.e., 
Pentium I, 133 MHz, 4mb RAM) which caused some functions not to work properly, 
most notably the expert graphing option. At the completion of the task list, participants 
viewed all portions of the prototype system, and formed an opinion on its effectiveness. 


Participants then completed an exit survey composed of demographic background 
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questions and perusal of the prototype system. Surveys were all submitted through a 


drop box provided in a common area. 


D. DATA TABULATION 

The data was transcribed from the survey onto a Microsoft Excel 2000 
spreadsheet. The Likert questions, based on a five-point scale, were coded into Excel, 
using numbers 1 through 5 corresponding to the respective anchors (Strongly Disagree, 
Disagree, Neutral, Agree, and Strongly Agree). Descriptive statistics were generated 
using Excel functions including the mean, standard deviation, range, and frequency 
distribution of the collected data. Content analysis was conducted on the responses 
provided from the open-ended survey questions. The categorization of participants by 
participant aircraft maintenance organization type and computer/software application 
experience level were noted. However, due to no noticeable differences between 


categories, all subsequent analysis was performed on all participants as a single group. 


E. DATA ANALYSIS 

Basic and general information about the demographic and question results were 
depicted using descriptive non-parametric analysis 1s conducted on the survey data in 
order to. Basic summary statistics are developed with results including demographic 
information and satisfaction levels with the prototype. Analysis of the data is performed 


using the functions of Microsoft Excel. 
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IV. RESULTS 
A. SAMPLE 
The 13 item exit survey was administered to 45 participants from a School of 
Aviation Safety “Aviation Safety Officer” course with a response rate of 95.6% (n=43). 
Each Naval Aviation command is required to have an officer trained by the Safety 
School. Thus the participants were designated Naval Aviators and Naval Flight Officers 
and represented a cross section of the aviation commands that make up the squadrons in 


the Navy and Marine Corps. 


B. DEMOGRAPHIC INFORMATION 

The material collected in Part I of the exit survey consisted of demographic 
information and established the aviation and computer experience levels of each 
participant had both with computers and in aviation. The information is later used to 
determine if experience level in either category affected a participant’s level of 
satisfaction and/or impacted the usability of the prototype MEIMS tool. The following _ 
paragraphs characterize the survey results for part I. 

Question one revealed that 39 of the Siem were members of aviations units 
that performed maintenance at the squadron level (n = 39, 90.7%). The remaining four 
participants were either members of higher-headquarter staffs (n = 2; 4.7%) or units that 
used civilian contract personnel to perform the maintenance (n = 2; 4.7%). Question two 
indicated that all but one participant (n = 42, 97.7%) stated they had at least two years of 
experience using acomputer. The remaining participant had less than one year of 


computer experience. Question three determined that all participants (n = 43; 100%) 
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were users of Microsoft Office, while minimal numbers had used either Lotus Notes (n= 
4; 9.3%) or Corel/Word Perfect (n = 3; 7.0%). Question four established a participant’s 
familiarity with different software applications, greater than 80 percent stated they were 
familiar with at least one of the following: word processing, spreadsheet, presentation, 
and e-mail (n >= 35; 81.4%). The average participant was familiar with approximately 
four (4.3) of the categories (see Table 4). 


Table 4. Number of Participants Familiar with Specific Software Applications 
(n=43) 


Processing Sheet ftware 


FFaniier[ 2 [37 
a 


Question five revealed the normal operating system (OS) for participants, 42 






(n = 42; 97.7%) responded with either Windows 97/2000, Windows NT, or both (see 
Table 5). Two participants did not answer the question. The prototype was loaded on 
computers running the Windows NT operating systems. Participants could indicate more 
than one OS. 


Table 5. Normal Operating System of the open (n=41) 


WindowsNT[ Mac [Unix | Linux 
NomalOSH 35 9-3 ps 
| % [sa | 707 | 73 | 99 [49 











54 


C. PARTICIPANT SATISFACTION WITH THE PROTOTYPE MEIMS TOOL 

(1) Responses to Impressions and Usability Questions 

Part II of the exit survey examined a participant’s impressions of the usability of 
the prototype MEIMS tool and its value to Naval Aviation. Participants responded to 
five statements using Likert type responses selecting from one of five responses: strongly 
agree, agree, neutral, disagree, and strongly disagree. Values of five through one 
respectively were assigned to the statements. One participant did not respond to any of 
the statements. The participants were also given the chance to make subjective 
comments on any of the five statements. 

(a) Statement one asked whether or not a participant found the prototype to 
be presented in a logical form. The histogram of the frequency distribution for statement 
one is presented in Figure 9. The mean was 4.26, standard deviation = 0.665, range = 4. 
Most participants (n = 39; 92.7%) agreed that the prototype was designed and presented 


in a logical fashion. One participant did not select a response. 


"Is in a logical format" 


a 
Responses 


E 
Strongly Agree Neutral Disagree Strongly 
Agree Disagree 





Figure 9: Exit Survey, Part II, Statement One, Response Distribution 


2) 


(b) Statement two asked about the ease of navigation of the prototype. The 


histogram of the frequency distribution for statement two is presented 1n Figure 10. The 


mean was 3.95, standard deviation = 0.947, range = 5. A large majority of the 


participants (n = 33; 80.5%) agreed that the prototype was easy to navigate. Two 


participants did not make a response to this statement. 


"Is easy to navigate" 


25 
20+ 


15+ 
# Responses 


Strongly Neutral Disagree Strongly 
Agree Disagree 





Figure 10: Exit Survey, Part II, Statement Two, Response Distribution 


(c) Statement three. The participants were asked whether they felt 


MEIMS was “interesting.” The histogram of the frequency distribution for statement 


three is presented in Figure 11. The mean was 4.07, standard deviation = 0.818, range = 


4. A large majority of the participants (n = 33; 80.5%) indicated the prototype was of 


interest to them. Two participants did not select a response. 
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Figure 11: Exit Survey, Part II, Statement Three, Response Distribution 
(d) Statement four asked about the relevance of the prototype to aviation 
maintenance operations. The histogram of the frequency distribution for statement four 
is presented in Figure 12. The mean was 4.40, standard deviation = 0.627, range = 3. 
Most participants (n = 39; 92.9%) indicated the prototype was of extreme relevance to 


maintenance operations. One participant did not respond to statement four. 
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Figure 12: Exit Survey, Part II, Statement Four, Response Distribution 


oy 


(e) Statement five asked whether prototype concept was a good one. The 
histogram of the frequency distribution for statement five is presented in Figure 13. The 
mean was 4.71, standard deviation = 0.427, range = 2. All participants (n = 42; 100%) 
indicated the concept of the prototype was a good one. One participant did not respond 


to this statement. 


"Is a good concept" 
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Figure 13: Exit Survey, Part II, Statement Five, Response Distribution 


(2) Responses to Open-ended MCT NE 

Part III of the exit survey contained three open-ended questions for the 
participants to respond to their overall satisfaction with the prototype. Every participant 
availed himself of this opportunity to provide constructive criticism. The responses from 
all 43 participants were overwhelmingly positive. Every participant indicated there was 
great merit in a tool such as the prototype and all of the “criticisms” were presented in a 
professional/positive manner. The desire of the participants was to take this prototype, in 
its current form, and improve it for their use in the fleet. 

(a) Question one asked the participant to list the most positive aspects of 


the prototype. Nine participants indicated the prototype was an excellent source of data 
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that could be used for training, trend analysis, and decision making. Others thought the 
prototype was useful to provide comparisons between variables (aircraft, mishap type, 
location, etc.). Some sample inputs include: 
e “Recurring maintenance issues stick out like a sore thumb.”’ 
e “MEIMS provides the ability to determine common mishap causal factors and 
prevent future ones of the same type.” 

e “MEIMS can help us look at our highest risk maintenance working 
conditions and identify those areas where we should be especially 
cognizant of potential disaster.” 

(b) Question two asked for the most negative aspects of the prototype. 

Overall comments indicated that participants with lower than normal computer “savvy”’ 

found it initially more difficult to navigate and understand the operation of the prototype. 

However, as interaction time with the prototype increased, so did the ease of operation. 

Problem areas of the prototype application were focused in one of three areas: HFACS- 

ME terminology, interface, and data entry. 

(1) HFACS-ME. Ten participants noted the HEFACS-ME 
taxonomy is not an ingrained part of everyday terminology and thus found it difficult to 
understand. The ability to access the HFACS-ME Code definitions from various parts of 
the prototype helped, but additional explanation of the each vice a mere translation of the 
three-letter acronym would have added more value to the participant. The participants 
felt that any eventual end-user of the prototype would need a good working knowledge of 


HFACS-ME in order to be able to get the most use out of the prototype; and that even the 


training received as part of the study may not be sufficient. 
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(2) Interface. Seven participants declared the prototype to not 
intuitively obvious in its operation. The fear was potential end-users with a lack of 
general computer skills would be discouraged from using the prototype. However two 
participants commented that the prototype became more user friendly with each use. Six 
participants said there was not enough on-line help available for usage training. 

(3) Data Entry. Though there were only four negative comments 
about data entry, they were all astute observations made by participants who obviously 
had advanced computer skills (though they were indistinguishable from others based on 
their demographic inputs). Comments on data entry included not having positive closure 
when data is entered and being able to enter the same data twice with no 
penalty/feedback. The remaining two comments were focused on unclear procedures for 
data entry. 

(4) Other “negatives’’. 

e Navigation issues were minor, limited to suggestions for improved access 
between pages (being able to go directly from one page to another without 
having to back out of previously selected pages-four participant inputs). 

e Ifthe participant selected parameters for a desired function (graphing or 
report) that were so specific that no data matched them, the function appeared 
not to work. The “error” message displayed to the participant did not 
satisfactorily explain the problem. 

e Insome instances the three-dimensional graphs in the front hide data in the 


back. Also, the graphs did not fully define the “colored” categories. 
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e Computer quality. There were several instances of one or more of the 
functions, especially in the graphing option, not working due to lack of 
memory in the computer. 

(c) Question three asked for suggested changes to the prototype. The 
participants brought out several key points critical for inclusion in future variations of 
MEIMS. Most of the suggestions related directly to one or more of the previously 
mentioned “negatives.” Nine comments were made about improving the ability for the 
end-user to understand HF'ACS-ME through either improved HFACS definitions within 
the prototype, additional help/tutorial on-line, and formal training for all end-users. Eight 
participants also made suggestions to improve the interface and navigation of the 
prototype to increase usability (e.g., adding additional methods to view HFACS-ME 
definitions and better descriptions of Levels 1, 2, and 3). Though no specific comments 
were made about data entry improvement, the “negatives” mentioned above imply fixes 
to be made: providing positive feedback Tron entering data, not being able to enter the 
same data more than once, and making the data entry procedures simpler and more clear, 

A noteworthy input made by four individual participants was in the area of “target 

end-user.” The onginal intent of MEIMS was for it to be distributed to squadrons for use 
in both data retrieval and data entry. Four participants made strong statements to the 
effect that the squadron level end-user should not be able to input data into the system, 
but that it should be done at a higher level, such as at the Naval Safety Center where the 
understanding of HEACS and HFACS-ME is greater and thus so is the ability to correctly 


input data into MEIMS. 
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Other inputs: 


Four participants indicated their desire to have more information about 
each mishap available for viewing (e.g., long summary vice short 
summary, adding the narrative of the mishap, etc.). 

Increasing the size of the data base by using mishaps prior to 1989 and 
using hazard reports was felt to be a means of improving the quality of the 
data (three participants). 

Eight specific changes to the actual interface were also suggested (e.g., 
increasing text box size in order to view all of the data field, a better 
method to show aircraft model to prevent confusion by adding the 
nickname to the model number: EA-6 Prowler, E-6 Mercury; being able to 
scroll through the chronological report vice viewing it page by page; 
separating H-1 into AH-1 and UH-1 categories, etc.). 

Using a higher speed, larger sete FY improved processor computer was 


also suggested to improve efficiency of MEIMS. 
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V. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 

A. SUMMARY 

Naval Aviation has determined to reduce its mishap rate. The reduction of human 
error involved in maintenance related mishaps will be one step in achieving that goal; 
now it has to find appropriate tools to accomplish this. The Human Factors Analysis and 
Classification System-Maintenance Extension (HFACS-ME) is a taxonomy which covers 
maintenance operations and falls in line with the Naval Aviation Safety Program’s notion 
of multiple causal factors, the idea of sequential events leading to an event, and several 
established human factors theories. HFACS-ME has been successfully used to examine 
human error in mishaps and incidents. The prototype MEIMS (Maintenance Extension 
Information Management System) tool is a safety information management system based 
on the HEACS-ME taxonomy used to facilitate the characterization and analysis of 
human error in Naval Aviation maintenance mishaps. Tools such as a final version of 
MEIMS will provide assistance 1n identifying human error patterns and facilitate 


intervention development. 


B. CONCLUSIONS 

The participants’ overall satisfaction of the prototype MEIMS tool indicated there 
is a need to provide access to mishap data information for use in training, analysis, and 
investigations. Participant feedback demonstrated the concept of MEIMS to be sound 
and its tie-in with Maintenance Operations readily apparent. However, the prototype 
requires some adjustment before successful implementation by end-users can be 


achieved. 
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The prototype, itself, received slightly lower, though still very positive, ratings 
than the concept of a maintenance human error related information management system. 
Even though a tool can be of good intent, if it is not considered usable by the end-user, it 
will sit on the shelf. For MEIMS to be “the tool” it must have its shortcoming resolved: 

e Lack of general maintenance organization HFACS-ME training including 
familiarity with HEACS-ME terminology.. 

e Less than desirable human-computer interface for end-users with below 
average computer skills. 

e Poor data entry confirmation indications. The inability for the end-user to 
enter data into the prototype in a simple and consistent format may lead to 
inconsistent inputs and hence poor data (1.e., garbage in, garbage out). 

e Also several minor shortfalls need to be refined: 

e Lack of standardized and convenient navigation. 

e Poor “error” messages in the cases of null data selections. 

e Some three-dimensional graphs hiding data depending on the view selected. 

e The inability to run the prototype successfully on some older personal 
computers. 

Providing solutions to these identified failings will 1mprove the usability of future 
versions of MEIMS; and subsequently the opportunity for it to be a factor in reducing the 


aviation mishap rate 1s enhanced. 
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C. RECOMMENDATIONS 
(1) Recommended Prototype MEIMS Tool Improvements 

e HFACS-ME. Incorporate improved HFACS-ME definitions within 
MEIMS by ensuring access to the definition page 1s available _ every 
form. Better descriptions of the HEACS-ME acronyms would also 
improve usability and understanding. Incorporating additional on-line 
help/tutorials will improve the end-users knowledge of HFACS-ME and 
make MEIMS a more productive tool for their use. 

e Interface. A computer science expert should participate in the fine tuning 
of MEIMS interface options to ensure navigation 1s consistent and easily 
done for those with sub-par computer skills. 

e Data Entry. Ensure data entry procedures are made as simple and clear as 
possible including providing positive feedback to the end-user once the 
entry has been taken. MEIMS must not allow repeat entries of the same 
data. 

e Target End-user. Unless data entry procedures can be significantly 
simplified MEIMS should be used as at the maintenance organization 
level in the read/analysis mode only. Entry of data should be conducted 
by higher levels (1.e., the Naval Safety Center for Naval Aviation) where 
the understanding of HFACS-ME 1s greater and thus so 1s the ability to 
correctly input data into MEIMS. 


e Include a longer summary/narrative for each mishap. 
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Additional data. Include mishaps pnior to 1989 and all hazard reports to 
improve the quality of the data base. 

Increase text box sizes to view all data field. 

Change aircraft identifier to include aircraft nickname in addition to 
type/model to avoid similar names. 

Separate AH-] and UH-1/ into two categories, vice only H-1 due to the 
aircraft’s inherent differences. 

Change the chronological report to a scrolling view, vice page by page to 
improve readability. 

If a selection is made for data that has a nul/ value, ensure the error 
message indicates the lack of response from MEIMS is due to “no data 
available for selected entry” vice simply an error with the system. 
Arrange data on three-dimensional graphs so that the fields with the 
largest numbers are put in the rear rows and scaled down to the front so 
that no data is hidden to the end-user. 

Suggested Computer Capability. Ensure end-users understand that 
computers with higher speed processors and larger memories will improve 
the efficiency of MEIMS. 

Add an option to include percentages on the three-dimensional graphing 
function, vice only quantity. This will show relative weight, vice always 
being more heavily weighted for aircraft types with a larger inventory 


(FA-18, H-46, etc.). 
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(2) The Future of MEIMS. The research for this paper involved data derived 
from the Naval Safety Center’s data base of Naval Aviation mishaps. A variation of the 
prototype MEIMS tool could be revised to include data from commercial mishaps (both 
passenger carriers and general aviation). Civilian aviation also has a record of human 


error, including maintenance related human error, contributing to mishaps. 
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APPENDIX A 


PROTOTYPE MAINTENANCE ERROR INFORMATION MANAGEMENT 
SYSTEM (MEIMS) TOOL REVIEW 


1. MAIN MENU 


The Main Menu app ears after MEIMS ICON is selected (see F gu y Al). 
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Select “Query Menu” lee button to view Query Menu (see Figure A2). 
2. QUERY MENU 
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Figure A A2: OSS Menu ft 
Select “Aircraft Model” command button. 
The Query by Aircraft Model menu appears (see Figure A3). 
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oer 





Figure A3: Query by Aircraft Model Sub-Menu 
Select aircraft model in combo box and select ““View Selection” command button. 


The ‘““Summary of Mishap” form appears (see Figure A4). 
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Select the see arrow after “Record:” on the bottom line to view additional records. 


F14 record number two appears (see Figure A5). Note that if the mouse arrow is placed 
over a text box or other control, a control tip text appears. Select “<Back” to return. 
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Figure AS: Summary of Mishap Form with Fi4 Selected as Aircraft Type, 
Record Two 
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Additional queries may be 
executed by selecting any of the 
control buttons on the left of the 
e Query Form (see Figure A6). 











io HFACS-ME Ramet 
| by ME Category using a 
Mukipie Crteria Query | 


eS 4 Retum to Main Menu 





Select Service 


< Back | 













JEmbarked 
1 Detached 
Unknown 


<Beck | 


E=| Select by Type 







Select Mishap Type 


Figure A6é: Query Menu with Sub-Menus for Additional Queries 


Select “Multiple Criteria” command button on Query Menu to more precisely define 
query (see Figure A7). a. 
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Figure A7: Query Menu with Multiple Criteria Sub-Menu 
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Define query by selecting desired check boxes and detailing information in combo boxes 



















(see Figure A8). 
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Select “View Sclecutths to view Summary of Mishap form (see Figure A9). 
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Figure A9: Summary of Mishap 
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command button to define Level 1, 2, and 3 codes (see Figure A10). 
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Figure A10: “Define HFACS Codes” Form 





This form may be selected at various points throughout the prototype in order to receive 
HFACS code definition. Return to Query Menu (see Figure Al 1). 
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Figure All: Ouer Menu 


Select HEACS-ME Summary command button. 


Ue. 


The HFACS-ME Summary form appears and displays HFACS [evel 1, 2, and 3 summary 
for all aircraft. Refine query by selecting desired information in combo boxes (see Figure 
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Return to Main Menu (see Figure A13). 


3. REPORT MENU 
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Select “Report Menu’. Report Menu appears (see Figure A14). 
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Figure Al4: Report Menu | 
Select “Mishap Distribution-All Mishaps” command button to view corresponding report 


(see Figure A15). 
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Figure A1S: Mishap Distribution-All Mishaps Report 
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Additional reports may be selected from the Report Menu including “All Mishaps- 
Chronological Listing” (see Figure A1l6). 


Chronological listing of all Maintenance Related 
Mishaps by Aircraft Type (1989-1999 
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60M 


Figure A16: All Mishaps-Chronological Listing Report 
Return to Main Menu (see Figure A17). 
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"Figure Al7: “Main Menu 
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Select categories for X X and Y axes. Make selections for X axis! 
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and for Y axis (see Figure A19). 
=} Multiple Selection 





Figure A19: Y-Axis Category Sub-Menu 
Select “Graph It” on Expert Graph Menu (see Figure A18). Three-dimensional graph 


appears (see Figure A20). 
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Figure A20: Expert Graph (Level 2 Codes vs Aircraft Type: EA6, F14, FA18) 


Return to Main Menu (see Figure A21). 
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, Figure AQ1: — == Menu — 


Select “Add New Data’”’. 


it 


Data Entry Form appears (see Figure A272). 
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Figure A22: Data Entry Form 
Enter data for new mishap (see Figure A23). Select “Add New Factor” for each factor to 
enter. When 3" Level Code entered, 2" and 1° Level Codes are automatically entered by 
the prototype. 
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Figure A23: Data Entry Form with Sample Data Applied 
Once complete, select “Close Form”. Final Data Entry Form appears (see Figure A24). 


Click on "Enter" box to complete data 
entry. Wait for check in box to appear 
before selecting "Close Form". - yee 


Es hee 





Close Form | 


Figure A24: Final Data Entry Form. | 
Select “Enter” to complete data entry. After check appears in “Done” box, select “Close 
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Form”’. 
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APPENDIX B 


PROTOTYPE MAINTENANCE ERROR INFORMATION MANAGEMENT 
SYSTEM (MEIMS) TOOL EVALUATION 


Background, Thank you for participating in a usability study (evaluation) of a 
prototype tool for the Maintenance Error Information Management System (MEIMS). 
This tool was developed by CDR Bnan Wood, USN as part of a thesis project for his 
Master of Science program in Information Technology Management. The information 
management system was developed to effectively address and identify patterns of human 
error in Naval Aviation maintenance-related aircraft mishaps. The Human Factors 
Analysis and Classification System Maintenance Extension (HFACS-ME) taxonomy is 
the foundation of MEIMS and is an effective method for classifying and analyzing the 
presence of human error in maintenance operations leading to major mishaps, accidents 
of lesser severity, incidents and maintenance related personal injury cases. However, 
working with a large database (approximately 600 Naval Aviation maintenance-related 
mishaps in Fiscal Years 90-99) is very labor intensive. Given the capability of current 
relevant database tools, an improved information management system will bring HFACS- 
ME to the next level. 


MEIMS captures maintenance error data, facilitates the identification of common 
maintenance errors and associated trends, and supports understanding of how to identify 
human errors in the future. The target audience for this information management system 
tool includes safety personnel (data entry & retrieval by unit safety officers, other safety 
& training personnel, maintenance officers, maintenance supervisors), mishap 
investigators-for data retrieval (Aircraft Mishap Board members, squadron safety 
officers), and analysts (from the Naval Safety Center, the command’s safety officer or 
one from its higher headquarters). A usability study demonstrated the effectiveness of 
the tool. This tool allows can directly lead to a decreased mishap rate and overall 
increased mission readiness due to the training and analysis opportunity it provides. 


Usability Study. Yu will be given a packet of instructions to guide you through 
MEIMS. You will be asked to make comments on the effectiveness and usability of the 
prototype system during your testing phase. Additionally, you will be asked to complete 
an “exit survey” after completion of your testing. Questions will include demographic 
information, objective questions about MEIMS usability, and subjective questions and 
comments for areas not covered in the objective section. The study should take no more 
than 15-20 minutes. 


Completion of Study. Upon completion of your testing and survey you will be 
asked to return your packet of instructions to CDR Wood’s office (E-305, East Wing 
Herrmann Hall). Pull this cover sheet off and put in the box marked “Cover Sheet.” Put 
the remainder of the survey (ensure stapled) in box marked “Remainder of Survey.” 


Thank you again, 
Brian Wood 
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Instructions for Prototype Maintenance Error Information Management System 
(MEIMS) Tool Evaluation 


Start-up 
1. Go to room E-300 (Computer Lab), E-320 (Ready Room), or E-322 (Computer Lab). 


Tum on computer (does not need to be logged into NPS LAN). 


Question 1: What is the name of your computer (aircraft name on CPU)? 


2. When Log-in menu appears, select <ESC> (this bypasses log-in requirement). 


3. When Desktop (main Icon screen) appears, double click (clicks are always with left 
mouse, unless otherwise stated) on “MEIMS” Icon. This will start the MEIMS 
application (in Microsoft Access 97). 


Main Menu 
4. You will now have the Main Menu displayed with the world famous Supersonic 


Hornet photo in the background. 


5. Note the five categories next to the command buttons on the bottom nght portion of 
the screen. The system has “focus” on “Query Menu”. Note the information on this 
button in the bottom left gray buffer above the Windows Start button. Place the mouse 
pointer over the Query Menu box (don’t click, if you do, select <Back> on subsequent 
page) and note information that appears in the Text Box (both of these sources of 
information will be available throughout MEIMS). 


6. Select <Tab> and view the same information for the remaining four command buttons 
(note, if you select <Exit> you will have to re-enter the system (see step 3 above). 


Question 2: Is the terminology clear enough to understand what each of the four 
command buttons does? If not, what could be changed to make it clearer? 


7. Select (click or tab to & enter) <Query Menu> 


Query Menu 
8. Note there are two sections on the Query Menu. The left half of the screen has seven 


categories to help you define how you would like to view the mishap data. The nght half 
of the screen has four command buttons. 


9. Select <Aircraft Model> 
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10. Another form appears: “Query by Aircraft Model”. Select your type aircraft, then 
select <View Selection>. “Summary of Mishap” Form appears. Note, your aircraft 
selection has a blue background. Review the “Brief Description” of the mishap and the 
“Contributing Factors.” View 

Question 3: What aircraft did you select? 


How many separate mishaps of that type aircraft are in the database? 


View one of the mishaps. 
What are the level 3 codes & what do they mean? 


How did you find that info? 


When you are through viewing the data, select <Close Form> 
Select another aircraft model (optional). 

When complete select <Back> on Query by Aircraft Model Form 
11. Select another category (your option) & view the data. 


Question 4: Which (if any) of the seven categories do you find useful? 


Which (if any) of the seven categories do you not find useful? 


12. Select <Multiple Cnteria>. Create your own query using two or more criteria. 


Question 5: Did you find this function useful? Why or why not? 


13. Return to Query Menu. Select <HFACS-ME Elements>. 
Question 6: How many total mishaps are in the database? 
How many mishaps have a level one category of Maintainer Conditions? 


How many mishaps have a level two category of Violations? 


14. Return to Query Menu. Select <HFACS-ME Summary>. 
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Question 7: How many total mishaps are in the database? 

How many mishaps have a level one category of Maintainer Conditions? 
How many mishaps have a level two category of Violations? 

Further define the system by your aircraft model (or select another type). 
Question 8: What aircraft did you select? 

How many separate mishaps of that type aircraft are in the database? 


Conduct further queries as desired. When complete, return to Query Menu & return to 
Main Menu. 


15. Select <Report Menu>. Review the six command buttons and their functions. 


16. Select <Mishap Distribution - Aircraft>. Find your type aircraft (or review another) 
in the report. <Close> the report & return to the Report Menu. 


17. Select <All Mishaps-Chronological Listing>. Review data. <Close> the report when 
complete & return to Report Menu. Return to Main Menu. 


18. Select <Expert Graph Menu>. Follow directions. Create one graph with aircraft 
model (yours and 1 or 2 others) on the X-Axis and HEFACS-ME Level One (all four 
codes) on the Y-Axis. 


Question 9: What aircraft did you select? 


Did you notice a difference in the level one codes between the aircraft (if so, what)? 


Return to <Expert Graph Menu>. Try more graphs as desired. When complete, return to 
Main Menu. 


19. Select <Add New Data>. Enter the following three mishaps to the database: 


Question 10: What is the Mishap Numbers for the data you are entering? 


Check to see if your entries were added to the database by Looking at the end of the 
Chronological Listing on the Report Menu (look for your Mishap Numbers). 
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Question 11; Did you see your data in the Chronological Listing? 


Return to Main Menu & Exit the Program. 


20. Please fill out the Exit Survey Questionnaire. 
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APPENDIX C 


PROTOTYPE MAINTENANCE ERROR INFORMATION MANAGEMENT 
SYSTEM (MEIMS) TOOL EXIT SURVEY 


User’s Impression of the Maintenance Error Information Management System (MEIMS) 
Prototype Tool 


Purpose: This survey evaluates a user’s overall satisfaction of the Maintenance Error 
Information Management System (MEIMS) prototype tool. It consists of three parts. 


Part I: Demographic Information. Part I provides the user’s aviation 
background, computer experience, and availability of software and hardware systems 
used in the Navy and Marine Corps. 


Part II: User Satisfaction with the Four Sections of the MEIMS Prototype Tool. 
Part II deals directly with user feedback as they use the prototype tool. 


Part UI: User Overall Satisfaction with the MEIMS Prototype Tool. Part If 
allows users to give general feedback about the prototype tool. 


Part I. Demographic Information 
Follow the instructions after each numbered question or statement. 


1. Iam attached to a command that primarily performs maintenance (military and/or 
civilian) at the: 
(Select one from the list and check the box) 


Squadron Level 

Intermediate Level (AIMD) 

Depot Level (NADEP) 

Command does not perform aircraft maintenance 
Other (describe if other) 


ODOBoOLD 


2. How long have you been using a computer? 
(Select one from the list and check the box) 


Less than one month 

One month to less than one year 
One year to less than two years 
Two years or more 


ao OD 
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3. What software do you normally use? 
(Check all boxes that apply) 


g Microsoft Office (Word, PowerPoint, Excel, Access) 
What version? 
(Check all boxes that apply) 
og 97 
2000 
not sure of version 
other (describe if other) 


OO O 


g Lotus Smart Suite (Word Pro, Lotus 123...) 
What version? 
(Check all boxes that apply) 
ap 
g 9.5 
g not sure of version 
gq other (describe if other) 


g Corel Word Perfect Office (Word Perfect, Quattro Pro...) 
What version? 
(Check all boxes that apply) 


Corel Office 7 

2000 

not sure of version 
other (describe 1f other) 


Ome oO OU 


g Other (describe if other) 


4. What software application categories are you familiar with? 
(Check all boxes that apply) 


Word Processing (MS Word, Word Perfect, Word Pro...) 
Spreadsheet (Excel, Lotus 123, Quattro Pro...) 
Presentations (PowerPoint, Harvard Graphics...) 
Graphic Software (Corel Draw, Adobe Photoshop...) 
E-Mail (Outlook, Eudora, AOL...) 

Database (Access, DBase...) 


Seo fe oO 
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5. What computer operating systems do you use? 
(Check all boxes that apply) 


Windows (3.1, 95, 98, 2000) 
Windows NT 

Macintosh 

UNIX 

Linux 

Other (describe 1f other) 


ES a ee 


Part II. User Satisfaction with the Four Sections of the MEIMS Prototype Tool 
Select the category that best matches your impression of each of the below categories 
(and check the box). 


Strongly Agree Neutral Disagree Strongly 
Agree Disagree 


I feel the information 0 O 0 7 7 
on the MEIMS tool was 
in a logical form 


(comments) 


I found the MEIMS 0 Ec 0 0 7 
tool easy to navigate 


(comments) 


My tour of the MEIMS ol O A a o 
tool was very interesting 


(comments) 

The information presented on 0 q f q q 
the MEIMS tool is relevant to 

maintenance operations 


(comments) 


The concept of the MEIMS O O 0 0 0 
tool is a good one. 


(comments) 
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Part III. User Overall Satisfaction with the MEIMS Prototype Tool 
Please make any comments on the MEIMS Prototype Tool not reflected in your 
comments in sections 1 and 2. 


The most positive aspects of the MEIMS prototype tool were: 


The most negative aspects of the MEIMS prototype tool were: 


J would make these changes (if any) to the MEIMS prototype tool: 


Thank you! Your participation is greatly appreciated! 
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